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19.  Abstract  (continued) 


interaction  with  the  aid  and  to  provide  prescriptive  guidance,  as  well  as  a  set 
of  interactive  dialogues  or  guides  which  blend  all  five  functions  in  support  of 
specific  user-selected  decision-making  strategies.  The  aid  will  be  tested  and 
demonstrated  in  the  context  of  a  personnel  selection  application. 
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1.0  INTRODUCTION 


1.1  Objectives 

The  overall  objective  of  this  project  has  been  to  develop  a  prototype  deci¬ 
sion  aid  that  successfully  blends  two  features:  (1)  it  is  personalized  in 
the  sense  that  it  accommodates  individual  differences  in  beliefs,  values, 
preferred  methods  of  problem  structuring  and  analysis,  preferred  methods  of 
organizing  information  and  searching  a  database,  and  variations  in  cogni¬ 
tive  "style,"  ranging  from  intuitive  to  analytical;  and  (2)  it  is  prescrip¬ 
tive  in  the  sense  that  it  attemp^s  in  a  variety  of  ways  to  steer  users  away 
from  potential  biases  or  errors  commonly  found  in  decision  making  and  as¬ 
sociated  with  preferred  decision  making  strategies.  During  Phase  I,  a  con¬ 
cept  for  such  an  aid  was  developed  and  demonstrated  in  the  context  of  a 
submarine  commander's  decision  as  to  when  to  launch  an  attack  on  an  enemy 
target.  This  decision  involves  a  difficult  tradeoff  between  waiting  to  ob¬ 
tain  a  more  accurate  fire  control  solution,  and  increasing  own  vul¬ 
nerability  to  enemy  attack.  The  attack  planning  situation  was  rich  enough 
to  allow  incorporation  of  a  wide  range  of  personalizing  options  selectable 
by  individual  users,  as  well  as  advisory  prompts  calling  the  user's  atten¬ 
tion  to  potential  pitfalls  or  inconsistencies  and  alerting  him  to  critical 
events.  This  demonstration  prototype  aid  was  described  in  the  Phase  I 
Technical  Report  (1982). 

The  original  objectives  of  Phase  II  were  as  follows: 

(1)  Complete  computer  implementation  of  the  design  concept 
developed  in  Phase  I . 

(2)  Design,  implement,  and  test  enhancements  of  the  original 
concept,  by  increasing  the  degree  of  personalization  and  the 
scope  of  the  coverage  of  the  advisory  prompts. 

(3)  Design,  implement,  and  test  a  general-purpose  (non-testbed 
specific)  personalized  decision  aid. 

Early  in  Phase  II  it  was  decided,  in  conjunction  with  the  ONR  Scientific 
Officer,  that  the  general-purpose  personalized  aid  would  take  the  form  of  a 
system  for  evaluating  and  selecting  options  from  a  large  database.  In  or- 
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der  to  stimulate  potential  commercial  Interest  In  the  development  of  the 
concept.  It  was  further  decided  that  the  database  subject  matter  for  Ini¬ 
tial  application  of  the  system  should  be  a  domain  of  widespread  Interest, 
and  the  problem  domain  of  personnel  selection  (from  a  large  database  of  ap¬ 
plicant  resumes)  was  chosen.  The  personalized  aid  Itself,  however,  Includ¬ 
ing  the  analytic  model  and  Interactive  features,  was  to  be  generally  ap¬ 
plicable  to  any  kind  of  option  evaluation  problem.  Finally,  In  order  to 
ensure  that  the  general-purpose  aid  had  the  highest  degree  of  personaliza¬ 
tion  and  prescriptive  capability  possible.  It  was  decided  to  focus  the  ef¬ 
fort  Involved  In  Objective  (2)  upon  the  general-purpose  aid  rather  than  the 
submarine  attack  planning  aid.  In  this  way,  the  end  product  would  have  the 
highest  probability  of  being  broadly  applicable  in  a  wide  variety  of 
decision-making  domains. 

1 . 2  Outline 

This  report  provides,  in  Section  2.0,  a  background  summary  of  the  research 
literature  describing  the  nature  of  commonly  found  cognitive  biases  in 
decision  making,  and  the  variations  in  problem-solving  strategies  charac¬ 
terizing  different  users,  or  indeed  the  same  user  at  different  times.  It 
thus  provides  a  rationale  for  the  personalizing  and  prescriptive  features. 
Section  3.0  describes  the  major  differences  between  the  attack  planning  and 
the  personnel  selection  decision  problems.  Section  4.0  describes  the 
general-purpose  aid,  highlighting  the  personalizing  and  prescriptive 
features.  Section  5.0  presents  conclusions  and  future  plans. 


2.0  BACKGROUND 


2 . 1  General  Description  of  the  Decision  Process 

The  decision-making  process  can  be  conceptualized  quite  generally  as  con¬ 
sisting  of  a  specific  set  of  cognitive  tasks  (Figure  1).  First,  goals  or 
objectives  must  be  known  or  identified  (if  these  are  not  present,  there  is 
no  motivation  to  decide  or  act).  Secondly,  current  circumstances,  insofar 
as  they  are  relevant  to  the  achievement  of  a  goal,  are  assessed.  If  a  dis¬ 
crepancy  is  perceived  between  goals  and  reality,  options  for  action  are 
generated.  If  more  than  one  option  is  available,  a  choice  will  be  made. 

This  is  by  no  means  a  rigid  sequence:  the  process  is  usually  iterative 
(for  example,  revising  goals,  reassessing  the  situation,  or  generating  new 
options  when  the  choice  process  falls  to  turn  up  an  acceptable 
alternative);  and  steps  may  be  skipped  (when,  for  example,  the  appropriate 
action  is  known  based  on  past  experience  with  very  similar  situations). 

But  the  basic  set  of  possibilities  is  as  shown,  at  least  in  many  of  the 
decision  contexts  we  have  considered,  and  some  such  framework  is  critical, 
we  believe,  for  identifying  the  specific  aspects  of  human  performance  where 
personalized  and  prescriptive  aiding  may  be  of  use. 

It  is  convenient  to  break  each  of  these  major  tasks  down  into  more  special¬ 
ized  cognitive  subtasks.  For  example,  situation  assessment  consists  of 
collecting  and  viewing  data  or  evidence,  deriving  inferences,  developing 
some  sense  of  confidence  in  the  conclusions,  and  continuing,  perhaps,  to 
draw  further  higher-level  inferences.  Again,  the  steps  may  be  iterative, 
may  be  combined,  or  may  be  skipped  altogether  by  ; ome  decision  makers  in 
some  situations. 

(Note  that  the  term  "evidence"  is  quite  relative;  evidence  in  one  process 
may  be  the  highly  uncertain  conclusion  of  a  prior  analysis.) 


2 . 2  Basis  for  Prescriptive  Features 


During  the  past  10-12  years  a  substantial  amount  of  research  has  Identified 
and  quantified  the  types  of  cognitive  biases  or  errors  commonly  made  during 
the  decision  process.  The  prescriptive  features  of  any  aid  must  be 
designed  to  prevent  or  counteract  these  types  of  biases,  to  the  extent 
possible . 

Each  of  the  cognitive  subtasks  identified  in  Figure  1  has  been  associated, 
at  least  in  laboratory  research,  with  characteristic  shortcomings  in 
reasoning.  Thus,  by  placing  recent  findings  in  cognitive  psychology  within 
this  framework,  we  may  derive  a  tentative  specification  of  the  types  of 
prescriptive  features  that  would  be  most  appropriate. 

The  following  summary  is  not  exhaustive;  it  is  meant  only  to  touch  on  some 
of  the  issues  that  bear  on  the  present  work.  Three  important  themes, 
however,  emerge:  (1)  Unaided  decision  processes  employ  simplifying  heuris¬ 
tics  that  at  best  only  approximate  prescriptively  accepted  rules  (e.g., 
Bayesian  probability  theory) ;  (2)  a  typical  effect  of  such  heuristics  is 
that  awareness  of  uncertainty  is  suppressed;  and  (3)  in  many  instances, 
biases  are  a  result  of  (otherwise  successful)  efforts  to  utilize  natural 
knowledge  structures  and  processes  of  reasoning. 

Assimilate  Evidence .  Patterns  of  information  search  in  laboratory  tasks 
tend  to  avoid  stringent  tests  of  favored  hypotheses  (Wason,  1960,  1981; 
Einhorn,  1980).  At  the  same  time,  there  is  a  tendency  to  seek  confirming 
evidence  of  an  already  well -supported  hypothesis,  rather  than  take  action 
or  consider  evidence  that  bears  on  other  issues  (Shaklee  and  Fischhoff, 
1982)  . 

Infer  Conclusions .  A  number  of  studies,  which  show  that  a  statistical 
model  of  a  person's  judgment  process  can  outperform  (in  accuracy)  that 
person's  own  judgments,  suggest  that  people  do  not  effectively  utilize  the 
information  available  to  them  in  inference  tasks  (Dawes,  1975;  Cohen, 
1982).  Other  laboratory  results  suggest  possible  causes.  For  example, 
people  tend  to  ignore  later  evidence  that  contradicts  a  favored,  or 


earlier,  datum  and  to  double  count  redundant  evidence  (Schum  and  Martin, 
1981).  Also,  people  commonly  ignore  statistical,  or  "base  rate",  data  and 
overweight  unique  or  problem-specific  factors  (Kahneman  and  Tversky,  1972). 
Both  of  these  observations  suggest  the  predominance  in  natural  reasoning  of 
non-statistical ,  causal  models  (Johnson,  1985).  Results  can  be  distorted, 
and  overconfidence  can  occur,  when  false  analogies  between  the  system  and 
the  model  influence  conclusions.  When  people  do  attempt  to  make  statisti¬ 
cal  judgments,  moreover,  estimates  may  be  biased  by  the  ease  of  recall  (or 
"availability")  of  a  particular  class  of  events  in  a  mental  sampling 
(Tversky  and  Kahneman,  1973). 

Assess  Quality  of  Conclusions .  A  number  of  studies  show  that  people  con¬ 
sistently  overestimate  their  degree  of  certainty  regarding  predicted  events 
and  estimated  quantities,  even  in  areas  where  they  are  (rightfully) 
regarded  as  experts.  While  there  is  some  evidence  that  experts  (as  opposed 
to  college  sophomores)  are  less  susceptible  to  overconfidence 
(Lichtenstein,  Fischhoff,  and  Phillips,  1982),  other  research  Indicates 
that  the  difference  between  expert  and  novice  is  slight  (Kadane  and 
Lichtenstein,  1982).  When  inference  proceeds  in  stages  (e.g.,  deriving  the 
probability  of  being  hit  by  enemy  fire  from  information  about  the  range  of 
a  threat,  which  is  derived  from  bearings  data),  people  often  simplify  the 
process  by  acting  as  if  conclusions  at  earlier  stages  (e.g.,  range)  were 
known  to  be  true,  rather  than  merely  inferred  (Schum,  DuCharme,  and 
DePitts,  1973).  Similarly,  the  probability  of  a  detailed  hypothesis  or 
scenario  is  likely  to  be  judged  higher  than  the  probabilities  of  its  com¬ 
ponents  (Tversky  and  Kahneman,  1983).  The  latter  effect  may  arise  because 
additional  details  increase  the  match  between  the  hypothesis  and  the  user's 
mental  models  or  knowledge  structures  (Leddo,  Abelson,  and  Gross,  1984). 

Option  Generation .  People  segment  complex  options  into  "natural" 
components,  and  treat  the  elements  as  if  they  were  independent  choices, 
leading  to  suboptimal  portfolios  (Tversky  and  Kahneman,  1981).  There  is  a 
tendency  to  formulate  options  in  terms  of  immediate  actions  that  span  only 
a  short  timeframe  rather  than  as  long-term  policies,  and  to  overlook,  as  a 
result,  the  cumulative  risk  of  pursuing  a  given  course  of  action  over  a 
long  period  of  time  (Slovic,  Fischhoff,  and  Lichtenstein,  1978).  In- 


divlduals  differ  in  the  degree  to  which  they  consider  future  choices  In 
current  planning  (Streufert  and  Streufert,  1981)  and  In  the  number  of  op¬ 
tions  they  generate  (Driver  and  Monk,  1976).  Ingrained  ways  of  viewing  a 
problem  tend  to  hinder  the  generation  of  novel  and  creative  solutions 
(Pitz,  Sachs,  and  Heerboth,  1980). 

Generate  Possible  Outcomes  of  Options.  In  considering  what  might  happen  if 
a  particular  option  is  adopted,  people  are  subject  to  biases  based  on  their 
internal  causal  models,  as  well  as  biases  in  recall,  such  as  a  heightened 
tendency  to  remember  salient  events  or  events  that  occurred  very  late  or 
very  early  in  a  sequence . 

Assess  Uncertainty  of  Outcomes .  Some  of  the  biases  which  affect  situation 
assessment  may  also  occur  when  predictions  are  made  contingent  on  a  par¬ 
ticular  option.  Additional  pitfalls,  however,  include  the  effects  of 
"wishful  thinking"  (e.g. ,  higher  probability  assessments  for  high  utility 
outcomes)  or  overcautiousness  (e.g.,  lower  assessments  for  high  utility 
outcomes).  According  to  Einhorn  and  Hogarth  (1984),  the  size  of  these  ef¬ 
fects  will  depend  on  the  degree  to  which  decision  makers  lack  confidence  in 
the  probability  estimates.  This,  in  turn,  may  depend  on  the  degree  to 
which  evidence  for  an  estimate  matches  the  type  of  evidence  represented  in 
user  knowledge  structures.  An  additional  set  of  biases  involves  distorted 
conceptions  of  randomness  in  everyday  judgment,  e.g.,  the  "gambler's 
fallacy"  where  a  sequence  of  similar  outcomes,  which  are  in  fact 
independent,  is  thought  to  increase  the  likelihood  of  a  different  outcome 
on  the  next  trial.  Fallacies  of  this  sort  may  be  inevitable  by-products  of 
powerful  top-down  or  expectancy-driven  processes  of  pattern  recognition 
(Lopes,  1982). 

Assess  Value  of  Outcomes.  Decision  makers  do  not  typically  consider  all 
the  potential  outcomes  of  an  action  together.  Rather,  outcomes  are  grouped 
into  "mental  accounts"  corresponding  to  natural  objects  or  causal 
relations,  and  choices  may  depend  critically  on  the  particular  grouping 
that  is  adopted  (Kahneman  and  Tversky,  1982).  An  additional  cognitive 
simplification  is  achieved  by  representing  an  outcome  in  causally  relevant 
terms,  by  the  difference  it  would  make  relative  to  some  reference  point. 
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Decisions  may  be  significantly  affected  by  the  choice  of  reference  levels, 
since  the  same  outcome  may  be  regarded  as  a  gain  or  as  a  loss.  For 
example,  the  outcome  of  a  defensive  tactic  may  be  encoded  as  400  men  saved 
(relative  to  the  number  who  would  have  died  had  nothing  been  done)  or  as 
200  men  lost  (relative  to  the  status  quo) .  An  Important  finding  by  Kah- 
neman  and  Tversky  (1979)  is  that  decision  makers  are  more  likely  to  take 
risks  when  outcomes  are  represented  as  losses  than  when  they  are  repre¬ 
sented  as  gains . 

Select  an  Option.  Heuristic  procedures  may  be  adopted  which  reduce  the 
cognitive  effort  that  would  be  required  in  a  thorough  consideration  of 
every  option.  Such  heuristics  have  implications  for  the  way  decision 
makers  search  information.  In  Elimination  by  Aspects  (Tversky,  1972),  for 
example,  search  is  organized  by  evaluative  attributes.  Attributes  are  con¬ 
sidered  serially  in  order  of  importance;  options  falling  below  a  cut-point 
on  an  attribute  are  eliminated  at  each  stage,  and  not  considered  further. 

In  this  strategy,  an  option  might  be  eliminated  for  missing  a  cut-point  on 
one  dimension  even  though  it  scores  very  highly  on  other  dimensions. 
Tradeoffs,  or  compensatory  relations  among  dimensions  are  thus  not 
considered.  In  another  heuristic  strategy,  called  "satisficing"  (Simon, 
1957;  Svenson,  1979),  information  search  is  organized  by  options.  The 
decision  maker  considers  a  sequence  of  options  until  he  finds  one  that 
clears  the  cut-points  he  has  selected  on  relevant  attributes.  Here  again 
compensatory  relationships  are  ignored.  Payne  (1981)  has  suggested  that 
these  information  search  strategies  may  correspond  to  the  way  decision 
makers  organize  knowledge . 

2 . 3  Basis  for  Personalized  Features 

How  are  the  users  of  decision  aids  likely  to  differ  in  their  approaches  to 
decision  making  and  problem  solving?  What  are  the  consequences  of  such 
differences  for  success  in  task  performance?  And  how  should  aids  be  per¬ 
sonalized  so  as  to  enhance  both  user  acceptability  and  quality  of 
performance? 
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We  consider,  briefly,  two  general  ways  In  which  decision  makers  have  been 
thought  to  differ  from  one  another: 

•  In  the  parameters  and  structure  of  a  prescriptive  model  based 
on  their  personal  beliefs  and  preferences;  and 

•  In  the  heuristic  strategies,  decision  processes,  and  cognitive 
styles  which  they  adopt  In  problem-solving. 

The  Interplay  of  findings  from  these  areas  helps  define  the  potentialities 
and  limitations  of  personalized  decision  aiding. 

2.3.1  Individual  prescriptive  decision  models.  Ironically,  a  driving 
force  in  the  evolution  of  prescriptive  theories  of  decision  making  has  been 
the  need  to  accommodate  individual  differences.  An  objective  rule  for  bet¬ 
ting  in  games  of  chance,  maximization  of  expected  value,  applies  only  where 
probabilities  of  outcomes  can  be  mathematically  defined  (as  in  rolling 
dice)  and  where  the  desirability  of  outcomes  is  physically  measurable 
(e.g.,  by  money).  Generalizations  of  this  basic  rule  to  situations  where 
those  conditions  do  not  hold  have  led  to  the  modern  technique  of  decision 
analysis  (cf . ,  Edwards,  1954,  1961;  Raiffa,  1968;  Brown,  Kahr,  and 
Peterson,  1974).  Von  Neumann  and  Morgenstern  (1947)  formalized  the  notion 
of  a  subjective  dimension  of  value,  i.e.,  utility,  and  extended  it  to  in¬ 
dividual  preferences  among  probabilistic  states  of  affairs.  De  Finetti 
(1937/1964)  and  Savage  (1954)  developed  formal  systems  for  the  quantifica¬ 
tion  of  an  individual's  "degree  of  belief",  or  subjective  probability, 
about  uncertain  propositions,  and  developed  axiomatic  justifications  for 
the  merging  of  utilities  and  subjective  probabilities  into  a  new  prescrip¬ 
tive  rule,  maximization  of  subjectively  expected  utility.  More  recently, 
rigorous  techniques  have  been  developed  for  combining  subjective 
preferences  with  respect  to  individual  components  of  value  into  a  single 
multiattribute  utility  measure  (e.g.,  Keeney  and  Raiffa,  1976). 

The  prescriptive  force  of  decision  analysis,  in  this  form,  is  not  to  dic¬ 
tate  to  an  individual  in  any  absolute  sense  what  he  "ought"  to  do  o^ 
believe.  Rather,  it  indicates  what  choices  and  beliefs  are  logically 


consistent  with  other  preferences  and  beliefs  which  he  chooses  to  accept 
(cf.,  French,  1979). 

These  elements  of  personalization  are  by  no  means  shared  by  all  prescrip¬ 
tive  approaches.  Techniques  In  operations  research  (e.g.,  cost/benefit 
analysis)  commonly  purport  to  be  "objective"  and  "value  free"  (Watson, 

1981).  The  approach  to  decision  analysis  described  above,  however,  has  two 
Important  Implications  for  personalized  aids: 

(1)  Dec Is Ion- analytic  aids  do  not  address  only  the  part  of  a  problem  that 
can  be  objectively  measured.  Actual  decisions  nearly  always  involve  a  num¬ 
ber  of  "soft  factors"  (e.g.,  uncertainty  about  the  intentions  of  a  business 
competitor  or  of  a  military  foe;  the  relative  importance  of  different 
objectives,  like  money  and  prestige).  The  decision  maker's  own  experience 
may  be  the  only  source  of  relevant  information  in  these  matters ,  while  an 
exclusively  "factual"  approach  could  be  fatally  incomplete.  Aids  which 
combine  subjective  and  objective  inputs  must  accommodate  individual  dif¬ 
ferences  among  users  in  assessments  of  uncertain  states  of  affairs,  at¬ 
titudes  toward  risk,  and  tradeoffs  among  competing  objectives. 

(2)  The  second  point  is  equally  important,  though  far  less  widely 
recognized.  Just  as  it  does  not  prescribe  inputs,  decision  theory 
constrains,  but  does  not  dictate  problem  structure.  Typically,  there  is 
more  than  one  way  to  express  the  probability  of  a  hypothesis  in  terms  of 
probabilities  for  other  propositions;  and  there  are  multiple  decompositions 
of  the  utility  of  an  option  into  preferences  for  separate  attributes .  A 
good  structure  for  a  particular  decision  maker  breaks  the  problem  down  into 
components  about  which  that  decision  maker  has  either  objective  data  or 
personal  experience.  Individuals  might  benefit  differently  from  different 
analyses  of  the  same  problem. 

In  particular,  it  has  been  suggested  that  experts  differ  from  novices  in 
their  capability  to  individually  recognize  a  very  large  number  of  different 
problem  situations  (De  Groot,  1965;  Chase  and  Simon,  1973).  Klein  (1980) 
argues  that  experts  tend  to  reason  holistically,  by  analogy  with  previous 
similar  experiences,  rather  than  by  explicit  analysis  and  computation. 
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Klein  warns  that  Imposition  of  analytical  models  may  actually  Impair  expert 
performance.  In  terms  of  decision  theory,  however,  this  distinction  be¬ 
tween  experts  and  novices  is  accommodated  by  the  notion  of  personalized 
problem  structures.  The  expert  might  produce  quite  creditable  holistic 
judgments  of  problem  components  which  he  has  "seen  before"  but  which  a  less 
experienced  individual  would  need  to  analyze  into  more  familiar  elements. 
(Nonetheless,  experts  too  are  subject  to  error- -particularly  when  a  problem 
which  appears  familiar  has  novel  aspects;  cf . ,  Sage,  1981.  Experts  may 
benefit  from  analysis  of  such  novel  components.)  The  implication  is  that 
if  decision  aids  are  to  exploit  the  capabilities  of  each  potential  user,  a 
variety  of  models,  with  different  functions  and  at  different  levels  of 
aggregation,  should  be  made  available  (cf . ,  Strub  and  Levit,  1974). 


2.3.2  Individual  strategies  in  inference  and  choice.  Prescriptive  deci¬ 
sion  theory  does  not  provide  a  description  of  actual  performance,  either  in 
probabilistic  reasoning  or  in  the  evaluation  of  actions  (cf . ,  Einhorn  and 


Hogarth,  1981).  Recent  research  in  cognitive  psychology  has  shed  light  on 


the  internal  processes  and  structures  which  people  employ  in  such  tasks , 


and  how  they  differ. 


One  line  of  research  has  explored  the  strategies  people  use  in  choosing 
among  actions.  Prescriptive  theory  requires  that  a  single  score  for  each 
option  (its  expected  utility)  be  derived,  which  integrates  all  the  avail¬ 
able  information  about  that  option:  i.e.,  its  score  on  each  of  a  set  of 
attributes,  or  the  probabilities  and  utilities  of  its  possible  outcomes. 
Several  descriptive  models  of  choice  behavior  have  been  proposed,  however, 
which  involve  more  partial  samplings  of  the  available  data  (e.g.,  Payne, 
1973;  Svenson,  1979). 


In  Tversky's  (1972)  Elimination-by-Aspects  (EBA) ,  for  example,  (as 
described  in  Section  2.2  above),  the  decision  maker  sequentially  considers 
each  attribute,  establishes  a  threshold,  and  eliminates  all  options  that  do 
not  score  at  or  above  the  threshold  on  that  attribute.  In  the  decision 
strategy  called  "satisficing"  (Simon,  1957;  Svenson,  1979),  the  decision 
maker  adopts  a  conjunctive  criterion  involving  cutoffs  on  one  or  more 
dimensions,  and  compares  successive  options  to  the  criterion  until  he  finds 
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one  that  Is  acceptable,  whereupon  he  stops.  These  different  decision 
strategies  have  different  Implications  for  the  order  In  which  people  elect 
to  receive  Information  (Payne,  1973,  1976).  Some  strategies  Imply  a  search 
organized  by  options,  others  a  search  organized  by  attributes. 

Individual  decision  makers  vary  In  the  decision  strategies  which  are 
reflected  in  their  information- seeking  behavior  and  in  their  verbal 
protocols  (Payne,  1976;  Russo  and  Dosher,  1981).  But  little  work  has  been 
done  to  discover  whether  these  individual  differences  are  consistent  across 
time  and  tasks  (Svenson,  1979);  instead,  emphasis  has  been  on  the  role  of 
task  variables.  For  example,  when  there  are  a  large  number  of  choice 
options,  decision  mckers  tend  to  select  routines  like  EBA  which  quickly 
eliminate  some  options  by  more  approximate  methods.  They  may  then  switch 
over  to  routines  which  integrate  all  the  available  information  about  the 
remaining  options  (Payne,  1976;  Wright  and  Barbour,  1977). 

Cognitive  style  has  been  regarded  as  a  relatively  invariant,  abstract  fea¬ 
ture  of  a  decision  maker's  approach  to  information  across  a  variety  of 
tasks  (cf . ,  Sage,  1981;  Libby  and  Lewis,  1977).  Perhaps  the  most  common 
differentiation  made  in  this  literature  is  represented  by  a  related  cluster 
of  distinctions  between  "analytic"  and  "heuristic"  (Huysman,  1970;  Mock, 
Estrin,  and  Vasarhelyi,  1972),  "abstract"  and  "concrete"  (Schroder,  Driver, 
and  Streufert,  1967;  Sage,  1981),  "systematic"  and  "intuitive"  (Bariff  and 
Lusk,  1977;  McKenney  and  Keen,  1974),  and  "scientific"  and  "managerial" 
decision  makers.  The  common  thread  is  a  distinction  between  preference  for 
formal,  explicit  analysis,  breaking  a  problem  down  into  elements,  and  an 
approach  based  on  global  intuition,  trial  and  error,  or  "common  sense". 

Unfortunately,  there  is  little  evidence  establishing  a  relationship  between 
these  categories  (based  on  self-descriptions)  and  actual  information¬ 
seeking  behavior  (Zmud,  1979;  Keen,  undated).  It  has  been  found  that  sys¬ 
tematica  generally  take  more  time  and  do  better  in  decision  problems  than 
heuristics  (e.g..  Mock  et  al . ,  1972).  Other  results,  however,  have  been 
inconsistent,  showing  that  systematics  prefer  more  information  or  less  in¬ 
formation  and  prefer  either  aggregated  or  raw  data  as  compared  to  heuris¬ 
tics  (cf . ,  Libby  and  Lewis,  1977;  Zmud,  1979).  McKenney  (quoted  in  Mock  et 
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al.,  1972)  states  that  the  propensity  to  be  analytical  Increases  with  task 
!  familiarity;  Klein  (1980)  and  Sage  (1981)  suggest  that  experts  will  be  more 

Intuitive. 


A  second  problem  In  this  literature  Is  the  failure  to  validate  the  claim 
that  cognitive  styles  are  task  invariant.  Studies  which  have  attempted  to 
do  so  have  produced  disappointing  results  (cf . ,  Libby  and  Lewis,  1977),  and 
recent  reviews  (Libby  and  Lewis,  1977;  Sage,  1981)  have  shifted  emphasis 
toward  the  influence  of  task  features  on  decision  styles  adopted  by  the 
same  individual  at  different  times.  Indeed,  Hammond,  et  al.,  (1984)  have 
shown  that  not  only  does  the  nature  of  the  task  Influence  decision  style, 
but  that  particular  ways  of  presenting  task-related  information  can  cause  a 
sh^  ft  toward  analytic  or  Intuitive  methods . 

In  a  few  cases,  "cognitive  styles"  have  been  defined  in  relation  to  actual 
cognitive  behavior.  Thus,  Driver  and  Mock  (1976)  defined  four  styles  by 
reference  to  two  fairly  specific  processing  dimensions:  amount  of  informa¬ 
tion  used  and  degree  of  focus.  The  latter  refers  to  a  tendency  to  consider 
only  one  solution,  model,  or  option  versus  a  tendency  to  entertain  multiple 
possibilities.  Streufert  and  Streufert  (1981a)  present  criteria  for 
"integrative"  decision-making  styles  in  terms  of  the  number  of,  and  length 
of  time  '’tween,  information  requests  and  decisions  based  upon  them. 
Streufert  and  Streufert  (1981b)  report  that  integrative  decision  making 
decreases  with  decision  urgency,  but  is  an  inverted-U- shaped  function  of 
the  amount  of  information  available. 


Implications  for  a  Personalized  and  Prescriptive  Aid 


Descriptive  work  on  human  inference  and  decision  processes  has  implications 
for  both  the  personal  and  prescriptive  aspects  of  decision  aiding. 


2.4.1  Personalization  and  efficient  flexibility.  "Flexibility"  in  and  of 
itself  is  not  a  sufficient  objective  in  system  design.  It  is  possible  to 
make  each  of  a  vast  number  of  logically  possible  information  acquisition 
strategies  equally  easy,  by  allowing  the  user  to  indicate  what  he  wants 
item-by- item.  But  such  a  system  does  not  really  facilitate  the  selection 
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of  strategies  as  such;  to  deal  explicitly  with  all  possible  search  orders 
would  be  beyond  the  time  and  capabilities  of  both  user  and  device.  The  ob¬ 
jective  of  personalization  Is  to  delimit  the  subset  of  strategies  which  an 
individual  Is  most  likely  to  prefer.  Decision  aids  may  then  be  tuned  to 
facilitate  explicit  selection  from  this  smaller  group  of  strategies,  while 
still  affording  the  general  "flexibility"  of  an  arbitrary  item-by-item 
search  sequence.  Such  aids  are  efficiently  flexible  in  their  responsive¬ 
ness  to  likely  user  needs. 

The  most  natural  way  to  acquire  and  process  information  can  vary  as  a  func¬ 
tion  of  the  individual  and  the  task.  Several  such  forms  of  variation  seem 
to  occur  frequently  enough  in  performance  to  justify  an  aid  design  which 
facilitates  their  employment: 

•  search  organized  by  options  or  by  attributes , 

•  decision  rules  based  on  cutoffs  or  tradeoffs, 

•  level  of  aggregation  of  information. 

In  addition,  it  seems  desirable  that  an  aid  facilitate  differences 
involving: 

•  focus  on  one  or  many  options, 

•  desired  amount  of  information,  and 

•  time  into  the  future  over  which  planning  takes  place. 

There  is  little  evidence  that  particular  individuals  are  consistent  across 
tasks  in  these  preferences,  and  some  indication  that  they  are  not.  In  the 
case  of  gross  categories  like  "intuitive"  and  "analytic",  moreover,  there 
is  no  reliable  mapping  of  traits  onto  system  design  features  and  certainly 
no  indication  of  how  different  traits  interact  (cf . ,  Huber,  1982). 

2.4.2  Prescriptive  aiding.  The  danger  inherent  in  complete  flexibility 
for  the  decision  maker  is,  of  course,  the  high  likelihood  that  one  or  more 
of  the  common  cognitive  biases  described  in  Section  2.2  will  result.  As  a 
safeguard  against  this,  two  types  of  prescriptive  aids  may  be  introduced: 
channeling  and  advisory  prompting.  The  difference  between  them  is  largely 
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one  of  tactics.  Channeling  Is  Implicit  and  proactive,  in  that  it  en¬ 
courages  users,  in  advance,  to  adopt  variants  of  their  own  preferred 
strategies  which  are  less  susceptible  to  biases  and  fallacies  of  Judgment, 
by  structuring  the  problem  in  such  a  way  that  those  variants  become  natural 
and  simple  to  execute.  By  contrast,  advisory  prompting  is  explicit  and 
reactive,  in  that  the  system  monitors  the  tasks  performed  by  the  human, 
identifies  steps  taken  that  are  likely  to  lead  to  error,  and  prompts  for 
the  addition  of  procedures  that  mesh  with  the  preferred  strategy  but  would 
minimize  the  possibility  of  error;  it  also  monitors  tasks  performed  by  the 
computer  and  prompts  where  a  human  contribution  might  improve  results. 

Thus,  in  advisory  prompting  the  computer  senses  weaknesses  in  a  line  of 
reasoning,  whether  its  own  or  the  user's,  and  offers  help.  Both  channeling 
and  advisory  prompting  may  be  viewed  as  examples  of  low-level  expert 
systems . 

Some  examples  of  how  the  prescriptive  aiding  techniques  of  channeling  and 
advisory  prompting  can  be  blended  into  a  personalized  system  are  given 
below. 

While  users  should  be  able  to  organize  displays  around  a  variety  of  mean¬ 
ingful  user -designated  objects,  the  aid  should  facilitate  the  use  of 
dec  is  ion- related  objects  for  this  purpose.  For  example,  channeling  can 
facilitate  clustering  of  options  by  their  performance  on  a  selected  evalua¬ 
tive  criterion.  When  an  intermediate  result  or  conclusion  is  uncertain, 
the  sources  of  its  uncertainty  should  be  explicitly  indicated.  Evidence 
for  a  result  should  be  available  for  display  along  with  the  result.  In¬ 
ferential  relationships  in  the  database  can  be  "mapped"  by  menus,  which 
permit  tracing  a  process  of  reasoning  from  its  sources  of  evidence  to  its 
final  conclusion. 

Research  in  cognitive  psychology  suggests  that  humans  tend  to  seek  addi¬ 
tional  confirming  evidence  for  a  favored  hypothesis.  An  advisory  prompt 
might  monitor  a  user's  pattern  of  information  requests,  examine  its  own 
model  of  the  problem  in  order  to  draw  inferences  about  the  hypotheses  the 
user  has  in  mind,  and  prompt  the  user  if  evidence  or  hypotheses  exist  which 
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the  user  has  failed  to  consider  but  which  may  have  an  Impact  on  his 
conclusions . 

Humans  often  find  It  difficult  to  assess  the  overall  credibility  of  a  con¬ 
clusion  based  on  several  steps  of  reasoning;  they  simplify  by  Ignoring  the 
uncertainty  at  early  stages.  Prompts  might  warn  users,  when  they  appear  to 
be  acting  as  If  a  particular  hypothesis  were  known  to  be  true,  that  a  num¬ 
ber  of  stages  of  uncertainty  must  be  kept  In  mind.  The  same  type  of  cau¬ 
tion  might  be  appropriate  when  a  compound,  or  conjunctive,  hypothesis  Is 
being  considered. 

The  user  might  be  notified  when  two  information  sources,  both  of  which  are 
regarded  as  credible,  have  contradicted  one  another.  He  might  then  choose 
to  readjust  one  or  both  credibility  assessments  downward.  An  advisory 
prompt  might  notify  him  on  future  occasions  when  either  of  the  (partially) 
discredited  sources  is  involved  in  an  important  conclusion. 

While  the  aid  should  permit  user  adjustment  of  any  meaningful  values 
employed  in  the  database,  channeling  should  selectively  facilitate  adjust¬ 
ment  of  values  about  which  users  are  likely  to  have  information  not  avail¬ 
able  to  the  computer.  Values  to  be  adjusted  could  be  decomposed  by  chan¬ 
neling  into  parameters  about  which  users  are  likely  to  have  reliable 
intuitions.  Automatically  computed  values  could  be  displayed  as  a 
reference,  so  users  can  focus  on  the  appropriate  direction  and  magnitude  of 
the  adjustment  (based  on  the  new  evidence)  and  not  have  to  integrate  all 
the  evidence  to  come  up  with  an  absolute  value. 

Humans  tend  to  combine  evidence  by  a  process  that  is  more  like  averaging 
than  like  proper  Bayesian  inference.  When  adjustments  fit  an  averaging 
pattern,  advisory  prompts  might  remind  subjects  to  consider  what  conclusion 
a  new  bit  of  evidence  favors,  before  performing  an  adjustment. 

Users  could  be  prompted  when  information  they  possess  may  be  of  significant 
value,  i.e.,  when  (1)  there  is  incompleteness  of  evidence  or  a  conflict 
among  lines  of  reasoning  in  the  computer  model  of  the  problem;  (2)  the  user 
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has  potential  access  to  relevant  Information;  and  (3)  the  result  is 
expected  to  have  an  impact  on  choices  among  actions  and  ultimate  payoffs. 

Channeling  could  facilitate  relatively  long  time  horizons  for  planning 
(e.g.,  by  displaying  appropriate  scaling).  Simultaneous  specification  of 
all  components  of  a  complex  option  could  be  facilitated  by  channeling. 
Displays  should  permit  generation  and  simultaneous  comparison  of  multiple 
options.  Channeling  should  facilitate  generation  of  options  which  include 
future  choices  or  contingencies. 

Short-range  planning  might  be  more  appropriate  in  some  situations  (e.g., 
where  feedback  is  continuous  and  mistakes  can  be  easily  and  quickly 
corrected),  while  long-range  planning  would  be  more  suitable  in  others 
(e.g.,  where  a  risk  appears  small  unless  it  is  considered  cumulatively  over 
the  long  run) ,  Advisory  prompts  might  recommend  that  the  user  consider  a 
shift  in  the  time  horizon  under  appropriate  circumstances. 

Users  should  be  prompted  if  they  have  generated  and  evaluated  a  complex  op¬ 
tion  piece -by-piece  and  if  overall  optimality  would  be  significantly  im¬ 
proved  by  considering  the  option  as  a  whole. 

The  user  should  be  prompted  if  only  one  option  has  been  considered,  but 
another  option  exists  which  is  superior  on  at  least  one  dimension. 

The  user  should  be  prompted  if  contingency  plans  have  not  been  incorporated 
in  an  option,  but  significant  new  information  is  likely  to  become  available 
during  its  execution. 

Channeling  could  draw  the  user's  attention  to  tradeoffs  between  different 
evaluative  dimensions  by  displaying  scores  for  an  option  on  more  than  one 
dimension  concurrently  (e.g.,  costs  and  benefits).  The  aid's  action  recom¬ 
mendations  should  be  explained  by  itemizing  how  options  differ  on  all  sig¬ 
nificant  dimensions.  Channeling  should  encode  and  display  outcomes  in 
terms  of  more  than  one  reference  point  (e.g.,  assets  lost,  assets  saved). 
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Humans  tend  to  employ  simplified  choice  schemes  that  disregard  tradeoffs. 

An  advisory  prompt  might  notify  the  user  when  he  has  eliminated  an  option 
because  it  fails  to  achieve  a  specified  level  on  a  favored  evaluative 
dimension,  if  that  option  has  significant  advantages  on  other  dimensions. 
The  user  might  be  told  how  much  stretching  of  his  specified  criterion  is 
required  to  readmit  the  rejected  option. 

An  advisory  prompt  should  occur  when  a  user  entertains  an  option  which  is 
dominated  (inferior  or  tied  on  all  dimensions)  by  other  options. 

While  users  should  be  free  to  designate  any  item  or  variable  in  the 
database  as  a  criterion  for  alerting,  alerts  should  also  occur  on  a 
prescriptive  basis.  Users  should  be  prompted  when  events  occur  or  facts 
are  learned  which  have  high  impact  within  an  appropriate  prescriptive 
model,  e.g.,  which  disconfirm  previously  well -supported  inferential 
hypotheses  or  which  significantly  affect  choices  among  actions. 

The  distinction  between  channeling  and  advisory  prompting  is  based  to  some 
extent  on  the  source  of  the  potential  bias  or  error.  Using  the  terminology 
of  behavioral  decision  theory,  if  "base  rate  data"  (i.e.,  a  body  of  re¬ 
search  findings)  suggests  that  errors  commonly  result  from  certain  ways  of 
representing  a  problem,  organizing  the  variables,  assessing  uncertainty, 
evaluating  outcomes  and  making  choices,  channeling  can  be  built  into  the 
aid  to  Increase  the  chances  that  the  user  will  adopt  amended  versions  of 
these  procedures  that  avoid  all  or  most  of  the  errors.  On  the  other  hand, 
if  "individuating  data"  (i.e.,  the  actual  procedures  and  judgments  of  the 
specific  user)  appear  to  be  moving  the  user  into  less -than- optimal  choices, 
advisory  prompting  can  explicitly  point  this  out  and  suggest  alternatives 
that  deviate  minimally  from  the  user's  originally  preferred  strategy 
(again,  preserving  the  user's  freedom  to  ignore  the  advice).  In  this  way, 
the  flexibility  provided  by  personalization  of  an  aid  can  be  tempered  by 
the  prescriptive  techniques  of  channeling  and  prompting  without  imposing 
undesired  constraints  on  the  user. 
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3.0  INTRODUCTION  TO  NEW  DECISION  MODEL 


As  indicated  in  Section  1.1,  the  prototype  aid  serving  as  a  context  for 
this  work  was  changed  from  one  dealing  with  a  submarine  commander's  deci¬ 
sion  about  when  to  launch  an  attack  on  an  enemy  target,  to  one  dealing  with 
a  personnel  manager's  '^election  of  a  new  hire  from  a  number  of  applicants. 
Aside  from  the  obvious  difference  between  the  two,  namely,  that  the  first 
deals  with  military  tactics  while  the  second  deals  with  a  broader  civilian 
(as  well  as  military  support)  decision  situation,  there  are  other,  more 
fundamental  differences  between  the  two  types  of  decisions  that  should  be 
pointed  out  before  the  new  system  is  described  in  detail. 

3 . 1  Degree  of  Model  Generality 

The  most  significant  difference  is  that  the  new  decision  problem  is  one 
that  lends  itself  to  a  much  more  generic  model,  applicable  to  a  wider 
variety  of  situations.  The  submarine  attack  model  is  typical  of  military 
tactical  situations  in  which  the  decision  maker  is  faced  with  a  clear 
tradeoff  between  firing  early  or  waiting  for  more  information  (which  may 
increase  the  probability  of  his  success  but  simultaneously  decrease  his  own 
survival  probability).  This  type  of  tradeoff  occurs  in  many  military  tac¬ 
tical  situations,  but  the  temporal  pace  of  the  action  and  the  factors  that 
must  be  built  into  the  model  vary  considerably.  These  tactical  models  must 
be  largely  tailored  to  specific  situations,  hence  their  generic  features 
are  severely  limited. 

The  personnel  selection  problem,  on  the  other  hand,  is  typical  of  a  wide 
variety  of  decisions  that  involve  multiple  evaluative  criteria  or 
objectives.  Thus,  if  properly  constructed,  the  system  can  be  made  ap¬ 
plicable  to  other  decisions  such  as  choices  among  political  candidates, 
policy  options,  investment  portfolios,  R&D  programs,  military  plans,  and 
many  others.  This  generality  can  be  accomplished  by  keeping  the  modular 
process  programs  entirely  separate  from  the  database.  Thus,  there  can  be 
modules  that  allow  the  user  to  build  his  own  preference  model,  to  ac¬ 
complish  word  processing  functions  while  building  the  model,  to  search  the 
database  in  various  ways,  to  organize  the  data  in  various  ways  for  analysis 


and  display,  to  enter  new  data  or  be  alerted  when  new  data  are  entered  by 
someone  else,  and  to  provide  channeling  and  advisory  prompting  prescriptive 
aids- -and  these  modular  programs  can  operate  on  any  type  of  data  that  are 
germane  to  the  specific  decision  problem. 

3 . 2  Real  Time  and  Stress 

Another  important  difference  between  the  two  decision  situations  is  that  in 
the  submarine  attack  decision,  real  time  plays  a  central  role  in  the  way 
the  problem  develops,  and  in  the  decision  itself,  and  contributes  to  the 
build-up  of  stress  in  the  situation.  In  the  personnel  selection  decision 
on  the  other  hand,  although  there  may  be  a  real  time  deadline  for  the 
decision,  the  process  itself  is  largely  self-paced,  under  the  control  of 
the  decision  maker,  and  stress  plays  a  negligible  role  in  the  situation 
(although  there  may  be  stressful  components  in  other  applications  of  this 
generic  model).  In  the  submarine  attack  problem,  the  firing  decision 
evolves  over  a  period  of  time  during  which  (in  most  cases)  more  and  more 
information  becomes  available,  and  information  seeking  is  always  an  impor¬ 
tant  option  to  be  considered.  In  the  personnel  selection  decision,  al¬ 
though  it  is  sometimes  possible  to  obtain  additional  information,  the  em¬ 
phasis  is  on  evaluating  existing  data  in  a  variety  of  ways  in  order  to 
select  an  option. 

In  the  submarine  case,  once  an  action  has  been  selected  and  performed,  the 
problem  essentially  is  re -set  and  may  begin  again  later  with  a  new  target. 
In  the  personnel  selection  case,  the  problem  often  continues  after  a  choice 
is  made,  since  multiple  choices  are  typically  possible.  Here,  subsequent 
choices  may  be  made  from  among  fewer  options,  or  new  options  (and  new  data) 
may  be  introduced  into  the  decision  situation.  Thus,  in  this  respect  the 
personnel  selection  decision  offers  a  richer  set  of  conditions  under  which 
the  decision  process  may  be  examined. 

3 . 3  Inference  vs  Choice 

Although  it  is  often  difficult  to  separate  the  two  components  of  inference 
and  choice  in  real-life  decisions,  there  are  certain  key  distinctions  be- 
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tween  them.  Inference  is  concerned  mainly  with  assessment  of  evidence  re¬ 
lated  to  the  relative  likelihoods  of  various  hypotheses  being  true,  and  the 
Judgments  being  made  are  usually  in  the  form  of  probabilities.  Although 
these  probabilities  usually  enter  into  the  choice  process,  the  distinctive 
feature  of  choice  is  the  assessment  of  preferences  regarding  the  various 
options  available,  or  the  predicted  consequences  or  outcomes  of  these 
options.  In  the  submarine  attack  decision,  the  more  significant  component 
was  that  of  inference  (about  enemy  position  and  intent) ,  while  in  the  per¬ 
sonnel  selection  decision,  the  more  crucial  component  is  that  of  preference 
assessment . 


3 . 4  Availability  of  Objective  Data 


In  the  submarine  context,  there  is  little  or  no  opportunity  for  the  user  to 
assess  and  adjust  the  validity  of  his  decision  model  by  reference  to  on¬ 
going  results  of  applying  the  model.  The  personnel  selection  decision,  on 
the  other  hand,  is  typically  much  richer  in  terms  of  its  database  of  on¬ 
going  results.  The  database  can  include  data  on  the  characteristics  and 
performance  of  current  and  past  employees  (in  addition  to  applicant 
characteristics),  so  that  the  user  can  in  fact  test  his  model  against  pre¬ 
vious  data  if  he  desires.  As  a  result  of  this  feature,  the  personnel 
selection  application  offers  a  large  set  of  opportunities  for  individual 
variations  in  data  organization  and  display.  Users  may  change  the  features 
being  considered,  their  importance,  the  sequence  in  which  they  are 
examined,  and  the  way  they  are  displayed,  as  well  as  the  level  of  detail  at 
which  the  analysis  is  conducted.  As  pointed  out  in  Section  2.4,  safeguards 
in  the  form  of  prescriptive  aids  must  be  provided  to  minimize  the  judgmen¬ 
tal  biases  that  could  emerge  as  a  result  of  this  flexibility,  but  the  over¬ 
all  aims  of  the  project  are  well  served  in  the  context  of  the  new  applica¬ 
tion  area. 


4.0  CURRENT  STATE  OF  SYSTEM  DEVELOPMENT 


W 


4.1  Hlffh  Level  System  Design 

The  conceptual  design  of  the  personalized  evaluation  system  has  been  or¬ 
ganized  around  two  closely  related  concerns:  (1)  Users  of  a  large  database 
of  options  may  differ  in  the  extent  to  which  they  know  what  they  want  or  do 
not  want,  in  the  naturalness  with  which  they  make  holistic  vs.  analytic 
judgments  of  preference,  in  the  decision  rules  and  information  search 
strategies  they  favor,  and  in  the  extent  to  which  they  seek  to  validate 
choices  by  examining  low-level  evidence.  (2)  In  the  very  experience  of  ex¬ 
amining  and  evaluating  options,  user  preferences  may  grow  increasingly 
determinate,  with  a  corresponding  shift  in  decision  and  information  search 
strategies.  As  a  consequence  the  system  is  designed  to  be  personal Lzed , 
that  is,  to  conform  to  a  user's  preferred  cognitive  style.  The  user  of 
such  a  personalized  aid  benefits  from  not  having  to  force  his/her  thinking 
into  an  uncomfortable  mold,  but  at  the  cost  of  relying  on  a  strategy  that, 
although  familiar  and  comfortable,  may  be  suboptimal.  To  guard  against 
this  risk,  the  aid  is  also  designed  to  be  prescriptive,  to  warn  the  user  of 
information  that  may  have  been  ignored,  or  errors  in  judgment  that  may  have 
occurred. 

The  system  is  built  upon  a  set  of  elementary  modules,  consisting  of  a 
knowledge  module,  four  cognitive  interface  modules  that  process  and  make 
changes  in  the  information  stored  in  the  knowledge  module,  and  a  fifth  cog¬ 
nitive  interface  module  whose  function  is  to  alert  the  user  when  actions 
taken  via  the  other  modules  may  differ  significantly  from  some  normative 
benchmark . 

The  experienced  user  can  access  these  modules  directly,  via  a  user-friendly 
interface  that  features  menu-driven,  mouse -controlled  graphical  displays. 
Users  may  also  wish  to  make  use  of  guides,  whose  purpose  is  to  step  the 
user  through  the  elementary  modules  in  accordance  with  one  of  a  set  of 
available  strategies  corresponding  to  different  cognitive  styles. 
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4.1.1  Overview  of  elementary  modules.  A  set  of  six  basic  modules,  inter¬ 
connected  as  shown  in  Figure  2,  form  the  basis  of  a  generic  decision  aid. 
Depending  on  specific  context,  the  Importance  attached  to  the  functions  of 
the  modules  may  vary.  Our  focus  in  this  report  is  on  an  evaluation  aid 
(specifically,  for  personnel  selection).  This  overview  describes  the 
generic  functions  of  each  module;  the  next  section  describes  how  each 
module  is  implemented  in  the  specific  context  of  personnel  selection,  and 
gives  examples . 


1.  Knowledge  Module  -  This  module  encodes  the  system's  knowledge  about 
the  decision  context.  Knowledge  includes  specific  information  about 
the  decision  problem  (e.g.,  options  and  facts  about  options),  general 
knowledge  about  the  problem  domain  (e.g.,  preferences  among  evalua¬ 
tive  criteria) ,  procedural  knowledge  about  how  the  decision  process 
is  to  be  carried  out,  and  knowledge  about  the  current  status  of  the 
decision  process.  This  knowledge  base  serves  as  input  for,  and  is 
altered  by,  the  cognitive  interface  modules. 

2 .  Adjust  Module  -  This  cognitive  interface  module  allows  the  user  to 
add  to  or  change  the  system’s  database  of  specific  problem  knowledge 
and  general  knowledge  about  the  problem  domain.  The  user  can  also 
create  or  alter  procedural  knowledge,  i.e.,  knowledge  about  how  the 
database  is  to  be  manipulated.  Inputs  may  be  specified  in  different 
ways  and  at  varying  levels  of  "fuzziness,"  depending  on  the  user's 
preference. 

3.  Decide  Module  -  This  cognitive  interface  module  allows  the  user  to 
apply  the  system's  procedural  knowledge  to  the  system's  database. 

The  user  may  specify  one  of  several  decision  making  strategies  or 
models  (e.g.,  multiattribute  utility  evaluation,  elimination  by 
aspects),  corresponding  to  different  cognitive  styles,  for  database 
manipulation  and  choice.  Evaluation  proceeds  to  the  degree  permitted 
by  the  level  of  specificity  of  user  inputs,  and  whatever  implications 
can  be  drawn  are  displayed. 

4.  Select  Module  -  Using  this  cognitive  interface  module,  the  user  may 
select  a  subproblem  on  which  to  focus,  a  subset  of  information  to  be 
displayed,  or  a  subcategory  of  the  database  to  be  adjusted. 

5.  Alert  Module  -  This  cognitive  interface  module  prompts  the  user  when 
events  occur  or  facts  are  learned  which  may  cause  significant  changes 
in  user  decisions.  This  function  is  most  important  in  real-time 
decision  aids,  when  rapid  assimilation  and  incorporation  of  incoming 
information  is  essential. 

6.  Advisory  Module  -  This  cognitive  interface  module  prompts  the  user 
when  he  or  she  appears  to  be  using  a  strategy  or  a  user -computer  task 
allocation  scheme  which  may  be  suboptimal  according  to  some  normative 
benchmark. 
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Figure  2:  Basic  Modules  of  Personalized  Decision  Aid 
(as  applied  to  Personnel  Selection  Aid) 


4.1.2  The  Guides.  A  guide,  as  Its  name  Implies,  has  the  function  of 
directing  the  user  through  the  basic  functions  in  accordance  with  a  given 
decision  strategy.  Thus,  each  guide  is  tailored  to  a  particular  strategy, 
and  the  cognitive  style  of  the  user  determines  which  guide  or  guides  (s)he 
chooses  to  invoke.  The  guides  will  in  general  be  adaptive,  enabling  the 
user  to  observe  the  implications  of  certain  inputs  and,  if  necessary,  cycle 
back  to  change  them.  Each  strategy  is  subject  to  its  own  set  of  possible 
biases,  and  an  important  function  of  the  guide  is  to  inform  the  user  when 
the  possibility  of  normative  violations  occurs. 

4 . 2  Prototype  Personnel  Evaluation  System:  Elementary  Modules 

As  discussed  in  Section  3.0,  the  personnel  selection  context  has  several 
distinguishing  characteristics  which  affect  the  design  of  a  decision  aid. 

The  aid  is  focused  on  choice  as  distinguished  from  inference.  In  addition, 
time  stress  and  the  incorporation  of  uncertainty  are  of  secondary 
importance,  while  data  organization  and  display  are  of  prime  importance. 

In  this  section,  we  discuss  both  the  elementary  modules  and  the  guides  as 
they  apply  to  the  aid  under  development.  The  aid  is  generic  in  that  it  can 
be  used  in  any  problem  domain  sharing  the  above  characteristics. 

4.2.1  Knowledge  module.  The  knowledge  module  consists  of  three 
components.  (1)  First  is  an  option  database  of  context-specific  knowledge. 
In  the  case  of  personnel  selection,  we  would  have  a  database  of  individuals 
(job  applicants)  and  their  relevant  characteristics  (salary  demand,  years 
of  experience,  etc.).  This  part  of  the  knowledge  module  is,  in  fact,  the 
only  aspect  of  the  system  that  is  specific  to  personnel  selection  as  dis¬ 
tinct  from  similar  choice  contexts.  (2)  The  second  component  of  the 
knowledge  module  is  a  preference  model  or  models,  and  an  associated  set  of 
decision  strategies,  which  capture  the  user's  knowledge  of  his  own 
preferences  in  the  problem  domain  and  his  or  her  preferred  method  for  using 
those  preferences  in  the  choice  process.  Preference  models  may  include 
numerical  weights  or  attributes,  intervals  or  ratios  of  weights,  rank  or¬ 
dering  of  weights,  cutoffs,  or  direct  evaluations  of  options.  Thus, 
preferences  may  be  specified  by  the  user  with  varying  degrees  of 
"fuzziness"  and  may  be  incomplete  in  some  or  many  respects.  User-selected 
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decision  strategies  may  be  based  on  multiattribute  utility,  elimination  by 
aspects,  satisficing,  or  dominance,  and  are  the  means  by  which  options  from 
the  database  are  retained  or  rejected.  (3)  The  third  component  of  the 
knowledge  base  is  Information  regarding  the  current  status  of  the  system. 

For  example ,  the  knowledge  module  will  keep  track  of  which  options  In  the 
database  are  "live"  and  which  have  been  eliminated  from  consideration  (via 
application  of  a  preference  model  by  use  of  the  Decide  module) .  In 
addition,  the  user  may  have  indicated  that,  although  the  model  is  specified 
in  terms  of  a  large  number  of  attributes,  (s)he  wishes  to  evaluate  options 
based  only  on  some  subset  of  attributes. 

4.2.2  Adjust  module.  This  module,  which  acts  on  the  first  two  components 
of  the  database,  has  two  major  functions.  The  first  function  is  to  allow 
the  user  to  make  changes  in  the  option  database  as  more  information  is  ac¬ 
quired  (e.g.,  a  candidate  is  no  longer  available,  the  performance  of  an  op¬ 
tion  on  an  evaluative  dimension  needs  to  be  changed  in  the  light  of  new 
information) .  The  second  (and  most  important  for  a  personalized  decision 
aid)  function  is  to  create  or  change  the  preference  models  by  means  of 
which  the  options  in  the  database  are  to  be  evaluated.  It  is  this  function 
that  can  most  significantly  be  tailored  to  individual  cognitive  styles. 

The  adjust  module  allows  the  user  to  specify  preferences  in  three  basically 
different  ways:  (1)  compensatory  knowledge  about  the  relative  importance 
(and  tradeoffs)  among  evaluative  criteria;  (2)  cutoff  levels,  i.e.,  non¬ 
compensatory  goals,  on  attributes  such  that  candidates  not  meeting  the 
cutoffs  are  rejected;  and  (3)  "bootstrapping"  or  "policy  capture"  judgments 
which  express  the  values  of  options  directly.  These  three  modes  correspond 
to  differences  in  the  degree  to  which  users  prefer  analytic  versus  intui¬ 
tive  and  concrete  approaches  to  choice.  In  addition,  as  noted,  analytic 
knowledge  in  mode  (1)  may  be  expressed  to  virtually  any  degree  of 
precision/imprecision  or  completeness/incompleteness. 

4.2.3  Decide  module.  The  function  of  this  module  is  to  apply  a  user- 
preferred  decision  strategy,  and  a  user-defined  preference  model  to  the 
database  of  options,  and  evaluate  and  display  the  results.  If  a  full  mul¬ 
tiattribute  utility  model  has  been  specified  (i.e.,  a  complete  set  of 


numerical  weights  on  all  attributes),  the  system  can  display  the  most 
preferred  alternative ( s) .  If  the  model  is  only  partially  specified  (i.e., 
user  Inputs  of  orderings,  intervals  and/or  ratios  of  weights  fall  short  of 
entailing  exact  values  of  weights) ,  the  system  displays  those  implications 
that  can  be  drawn  from  the  Information  the  system  has.  For  example,  the 
system  can  compute  the  feasible  alternatives,  i.e.,  those  options  that  can¬ 
not  be  eliminated  based  on  the  available  knowledge  of  the  user's 
preferences.  In  addition,  the  system  can  perform  a  type  of  dominance 
testing;  that  is,  given  one  alternative  (hypothetical  or  from  the 
database) ,  it  can  compute  which  alternatives  are  known  to  be  at  least  as 
good.  Finally,  the  system  can  perform  elimination  by  aspects,  a  choice 
strategy  requiring  only  an  ordering  on  the  attributes  and  cutoff  levels  for 
each  attribute . 

4.2.4  Select  module .  This  module  allows  the  user  to  select:  (1)  aspects 
of  the  preference  model  for  display  or  adjustment,  (2)  information  about 
options  for  display  or  adjustment,  and  (3)  a  subset  of  the  current 
preference  model  for  application  in  choice. 

4.2.5  Alert  module.  The  function  of  this  module  is  to  notify  the  user  of 
relevant  changes  in  knowledge  about  options.  This  module  is  of  primary  im¬ 
portance  in  real-time  systems  in  which  the  system  must  assimilate  informa¬ 
tion  about  the  outside  world  faster  than  it  can  be  attended  to  by  the  user. 
Such  a  system  must  help  the  user  to  cope  with  "information  overload"  by 
alerting  the  user  to  possibly  useful  new  information. 

Even  in  the  present  context,  however,  information  overload  may  plague  the 
user  of  a  large  database.  If  new  options  are  incorporated  into  the 
database,  or  changes  occur  in  the  data  regarding  old  options,  then  the  user 
might  be  alerted  if  (and  only  if)  those  changes  have  implications  for  the 
current  "best  alternative  set." 

A  second  mode  of  alerting  involves  user-specified  alerts:  for  example,  the 
system  can  be  told  to  provide  an  alert  if  a  candidate  shows  up  with  four 
years  of  programming  experience. 


4.2.6  Advisory  module.  This  module  functions  as  an  overseer  of  the  other 
modules,  checking  when  a  preferred  user  decision  strategy  may  result  In 
significantly  suboptlmal  choices.  (1)  Adjust:  For  example,  users  would  be 
prompted  when  inputs  provided  under  the  Adjust  module  are  significantly 
inconsistent,  and  ways  of  resolving  the  inconsistency  would  be  suggested. 

If  the  information  provided  by  a  user  is  too  imprecise  or  incomplete  for 
successful  application  of  the  preferred  decision  strategy.  Advisory  prompts 
would  suggest  additional  inputs  that  are  needed  (e.g.,  the  number  of  live 
options  could  be  narrowed  down  from  100  to  the  desired  10,  if  only  the  or¬ 
der  of  importance  of  education  and  experience  is  indicated) .  Users  might 
also  be  notified  if  small  changes  in  user  inputs  would  have  a  significant 
impact  on  choice.  (2)  Decide:  in  the  Decide  module,  users  applying 
elimination-by-aspects  or  satisficing  would  be  prompted  to  reconsider  an 
option  which  has  been  discarded  because  it  fails  to  meet  a  cutoff  on  one 
attribute,  but  is  outstanding  in  other  respects.  Conversely,  the  user 
would  be  notified  if  an  option  which  is  accepted,  because  it  clears  all 
cutoffs,  is  in  fact  dominated  (i.e.,  there  is  an  option  set  which  is  at 
least  as  good  as  that  option  on  all  dimensions  and  better  on  at  least  one 
dimension).  (3)  Select:  Users  would  be  prompted  when  user-selected  at¬ 
tributes  fail  to  capture  significant  variance  in  the  option  database.  (4) 
Alert:  Users  would  be  prompted  when  user-defined  alerting  criteria  fail  to 
capture  significant  aspects  of  the  current  preference  model. 

In  all  cases,  users  would  control  the  degree  of  significant  of  the  problem 
that  is  to  trigger  an  advisory  prompt.  Moreover,  users  are  free  to  accept 
or  reject  any  advice  that  is  offered, 

4 . 3  Current  Status 

This  section  describes  in  somewhat  more  detail  selected  functions  in  the 
personalized  evaluation  system  that  are  currently  implemented,  partially 
implemented,  and  planned. 


4.3.1  Adjust. 


4. 3. 1.1  Problem  structuring  (partially  Implemented) - -This  function  will 
allow  initial  structuring  of  the  problem,  i.e.,  identification  of  the  op¬ 
tions  to  be  considered  and  the  attributes  on  which  they  are  to  be 
evaluated.  The  user  will  be  able  to  organize  the  attributes  hierarchically 
using  interactive  graphics  for  "tree-building,"  so  that  analyses  performed 
at  any  level  will  be  aggregated  upward  (or  decomposed  downward,  if  desired) . 

4. 3. 1.2  Entering  compensatorv/tradeof f  information  on  attribute  weights 
(partially  implemented) - -The  user  can  enter  compensatory/tradeoff  informa¬ 
tion  on  the  relative  importance  of  attribute  weights  in  any  one  of  three 
ways  (or  in  any  combination  of  the  three) : 


•  The  Weights  Screen  (Implemented)  -  On  the  Weights  screen 
(Figure  3),  the  user  can  enter  upper  and  lower  bounds  for  each 
attribute  weight.  By  setting  the  upper  and  lower  bounds  equal, 
a  precise  weight  may  be  entered.  The  user  also  sees  the  bounds 
implied  by  all  other  judgments  (s)he  has  made  on  this  and  any 
other  screens.  Figure  3  shows  a  sample  Weights  screen  display, 
on  which  the  user  has  indicated,  for  example,  that  the  weight 
for  the  first  attribute  (Alpha)  should  be  between  35  and  60  on 
a  scale  from  0  to  100,  with  the  other  attributes  weighted  as 
shown. 

•  The  Ratio  Screen  (Implemented)  -  The  Ratio  screen  (Figure  4) 
allows  comparisons  of  relative  magnitudes  of  attribute  weights. 
One  of  the  attributes  may  be  specified  as  the  standard,  and 
other  attribute  weights  are  assessed  relative  to  it.  In  the 
sample  screen  of  Figure  4,  the  decision  maker  has  indicated 
that  the  Beta  attribute  should  be  given  weight  between  equal  to 
and  about  20%  higher  than  that  of  the  Alpha  attribute.  The  in¬ 
formation  is  coded  internally  into  linear  inequality  con¬ 
straints  on  the  attributes.  Again,  the  decision  maker  can  see 
the  implications  of  all  other  inputs  to  the  system  (on  this  or 
any  other  screen)  as  they  relate  to  relative  magnitudes  of  at¬ 
tribute  weights. 

•  The  Order  Screen  (Partially  Implemented)  -  On  this  screen  the 
user  can  enter  information  merely  about  the  order  of  importance 
of  attribute  weights  (e.g.,  that  the  first  attribute  should 
have  higher  weight  than  the  second  without  saying  any  more 
about  what  those  weights  are).  Once  again,  the  implications  of 
all  user  inputs  for  rank  order  of  weights  is  displayed. 
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Figure  3:  Weights  Screen 
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Figure  4:  Ration  Screen 


Each  of  these  screens  allows  a  user  to  think  about  his  or  her  preferences 
in  a  different  way.  Yet  they  are  highly  integrated,  since  the  implications 
of  all  currently  available  information  (entered  from  any  of  the  above 
screens,  or  from  the  bootstrapping  screen  described  below)  can  be  displayed 
on  each  screen.  The  mechanism  for  computing  implications  across  screens  is 
linear  programming. 

The  entering  of  information  on  each  of  these  screens  corresponds  to  build¬ 
ing  or  changing  a  preference  model. 

4. 3. 1.3  Cutoffs  on  attribute  scores  ( implemented) - -On  the  Cutoffs  screen, 
the  user  can  indicate  minimal  and/or  maximal  acceptable  levels  for  any 
attribute,  so  that  alternatives  not  meeting  the  cutoff  are  eliminated  from 
consideration.  Cutoffs  can  also  be  provided  using  the  Elimination-by- 
Aspects  function,  described  below. 

4. 3. 1.4  Bootstrapping  (planned) - -On  the  bootstrapping  screen,  the  user 
makes  direct  judgments  about  alternatives,  either  real  or  hypothetical. 

The  user  may  place  bounds  on  the  score  of  the  alternative  (e.g.,  "between 
65  and  80  on  a  scale  of  100")  or  may  make  a  direct  comparison  between  two 
alternatives  (e.g.,  "Jones  would  score  better  than  Smith").  As  for  the 
above  screens,  bootstrapping  information  can  be  encoded  as  linear  in¬ 
equalities  on  attribute  weights. 

Bootstrapping  is  an  indirect  way  of  giving  the  system  information  on  at¬ 
tribute  weights.  Hence,  the  Bootstrapping  screen  may  be  viewed  as  part  of 
the  Adjust  module,  and  implications  of  the  Bootstrapping  judgments  are 
stored  as  part  of  the  Knowledge  module. 

4.3.2  Decide  (partially  implemented).  Bv  pointing  to  "Go"  and  clicking 
the  mouse  key,  the  user  instructs  the  system  to  apply  whatever  knowledge 
about  his  preferences  the  system  has  gathered  to  the  current  set  of 
options.  Two  settings  are  provided  which  influence  the  performance  of  this 
function:  "#  Needed"  indicates  the  number  of  options  which  the  user  ul¬ 

timately  desires  to  choose;  it  may  range  from  1  to  the  number  of  items  in 
the  database  (a  trivial  choice  problem!).  The  first  time  "Go"  is  selected. 
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the  preference  model  is  applied  to  the  entire  database  of  options.  The  ap¬ 
plication  of  that  model  is  likely  to  result  in  a  smaller,  weeded  down  sub¬ 
set  of  options  which  remains  feasible  or  "live"  (i.e.,  a  set  of  options 
which  is  known  to  contain  the  desired  choices,  but  within  which  further 
discriminations  cannot  be  made  based  on  present  inputs).  The  user  may,  if 
he  wishes,  return  to  the  Adjust  screens,  provide  further  inputs  (or  revise 
old  ones),  and  then  select  "Go"  again.  Each  subsequent  selection  of  "Go" 
operates  on  the  currently  prevailing  set  of  live  options,  further  narrowing 
it  down.  The  user  has  the  option,  however,  of  resetting  this  process  by 
selecting  "Use  all."  In  that  case,  the  next  use  of  "Go"  operates  on  the 
original,  full  database. 

By  these  simple  commands,  the  user  can  implement  any  of  a  variety  of  choice 
strategies.  For  example,  by  setting  cutoffs  on  additional  dimensions  be¬ 
tween  each  use  of  "Go,"  he  may  successively  eliminate  options,  through  an 
elimination-by-aspects  strategy.  If  the  user  has  provided  information  on 
cutoffs  and  on  the  ordering  of  dimensions,  the  system  will  in  effect  per¬ 
form  an  el imination-by-aspects  analysis  for  him,  applying  the  cutoffs  on 
each  dimension  to  the  option  set  in  the  specified  (or  implied)  order  of 
importance,  stopping  if  and  when  the  option  set  is  reduced  to  the  desired 
number  of  choices.  Any  compensatory  information  will  also  be  used  directly 
by  the  system  to  eliminate  options.  The  result  of  "Go"  is  always  the  set 
of  feasible  options,  i.e.,  those  that  could  be  optional  given  current 
information. 

4.3.3  Guides .  In  addition  to  these  basic  functions,  a  variety  of  guides 
and  special  screens  support  the  decision  process.  These  involve  functions 
not  only  from  the  Decide  module,  but  also  from  Adjust,  Select,  and 
Advisory,  orchestrated  in  a  user-friendly  dialogue  that  reflects  typical 
decision  making  patterns. 

4. 3. 3.1  Eliminate  (partially  implemented) - -For  example,  instead  of  per¬ 
forming  elimination-by-aspects  by  means  of  elementary  commands  (as 
described  above) ,  the  user  can  receive  additional  guidance  through 
"Eliminate."  This  guide  directs  the  user,  in  a  highly  flexible  manner, 
through  the  following  steps; 


'38- 


(1)  Selection  of  attributes  for  Inclusion  in  the  choice  process  (by 
graphical  pointing) ; 

(2)  use  of  ordering  screen  to  rank  order  the  selected  attributes, 
with  prompts  to  elicit  from  the  user  a  single  connected 
ordering: 

(3)  use  of  cutoff  screen  to  elicit  cutoffs  for  the  selected 
attributes ; 

(4)  display  of  results  on  the  EBA  screen. 

These  steps  are  by  no  means  rigid.  For  example,  if  he  or  she  is  satisfied 
with  the  already  existing  attribute  selection,  ordering,  and  cutoff 
specification,  the  user  may  proceed  directly  to  step  (4) .  At  any  time 
thereafter,  the  user  may  return  to  any  previous  step,  make  changes,  and  ob¬ 
serve  the  results . 

The  EBA  screen  (Figure  5)  functions  as  an  elimination-by-aspects 
"spreadsheet."  It  displays  the  selected  attributes  in  order  of  importance, 
the  lower  and/or  upper  cutoffs,  and  the  number  of  surviving  options  at  each 
stage  in  the  process  (i.e.,  after  each  additional  set  of  cutoffs  has  been 
applied) .  In  addition,  an  advisory  prompt  is  provided  which  notifies  the 
user  if  there  are  promising  options  which  have  been  rejected.  These  are 
options  which  fall  outside  the  cutoffs  on  an  attribute,  but  score  very  well 
on  other  dimensions.  In  addition,  the  user  is  told  by  how  much  the 
specified  cutoffs  would  have  to  be  stretched  to  include  these  rejected  (but 
promising)  options. 

The  user  can  modify  any  cutoff  directly  on  this  screen,  and  the  EBA  will  be 
recomputed,  showing  which  alternatives  are  thereby  excluded  or  re-included. 

By  use  of  the  mouse  and  the  VIEW  option  (Figure  6) ,  the  user  can  "zoom"  in 
on  the  details  of  any  part  of  the  EBA  screen.  For  example,  the  user  can 
view  all  database  options  in  the  vicinity  of  the  lower  or  upper  cutoff  on 
any  attribute,  or  he  can  view  the  set  of  promising  but  rejected  options. 

The  reasons  for  considering  an  option  "promising"  are  also  indicated.  From 
the  zoomed- in  point,  the  user  can  scroll  anywhere  else  in  the  database  of 
options.  The  VIEW  screen  can  also  be  used  to  modify  cutoffs  in  a  way  that 
differs  significantly  from  the  CUTOFFS  screen  or  the  EBA  screen.  The  user 


Figure  5:  EBA  Screen 
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may  graphically  move  upper  and  lower  cutoffs  against  the  context  of  actual 
options  to  be  Included  or  excluded,  rather  than  merely  against  an  abstract 
numerical  scale . 

4. 3. 3. 2  Justify  ^partially  Implemented! - -The  Justify  screen  (Figure  7) 
provides  users  with  a  comprehensive  evaluative  overview  of  selected 
options.  The  objective  Is  to  support  the  final  stages  of  a  decision 
process,  In  which  users  seek  to  examine  candidate  choices  In  detail,  to 
look  for  previously  overlooked  weak  points,  and  to  construct  a  Justifica¬ 
tion  for  the  choice.  The  Justify  screen  for  a  given  option  displays  the 
score  of  that  option  on  each  attribute,  the  best  score  of  any  option  on 
each  attribute,  and  the  difference  (in  standard  deviations)  between  the 
current  option's  score  and  the  best  score;  it  also  shows  how  many  options 
scored  better  than  the  present  one  on  each  attribute  and  what  percentage  of 
the  total  database  they  represent.  Finally,  it  indicates  if  the  present 
option  is  poor  or  outstanding  on  any  particular  dimension. 

A  VIEW  option  is  associated  with  Justify  which  enables  users  to  "zoom"  in 
on  and  examine  in  detail  the  options  which  are  superior  to  the  given  option 
in  any  particular  dimension. 

For  some  decision  makers,  the  Justify  screen  might  play  a  central  role  in 
decision  making.  The  process  of  justifying  a  choice  may  lead  to  recon¬ 
sideration  and  revision  of  previous  preference  judgments.  Montgomery 
(1983)  has  argued  that  decision  making  in  general  is  a  search  for  good  ar¬ 
guments  or  justifications.  A  characteristic  strategy  is  to  accept  an  op¬ 
tion  as  justified  only  if  efforts  to  create  a  dominance  structure  for  that 
option  are  successful,  i.e.,  when  preferences  can  plausibly  be  represented 
in  such  a  way  that  the  option  appears  as  good  or  better  than  other  options 
on  all  dimensions.  A  stronger  justification  occurs  when  the  preferred  op¬ 
tion  is  shown  to  be  uniquely  outstanding  on  at  least  one  dimension.  The 
Justify  screen  enables  users  to  ascertain  quickly  whether  requirements  of 
this  sort  are  satisfied.  If  not,  users  may  alter  the  representation  of 
their  preference  model  until  dominance  applies,  e.g.,  by  combining  at¬ 
tributes  (via  the  problem  structuring  screen) ,  by  reducing  the  importance 
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Figure  7 :  Justify  Screen 


of  attributes  on  which  the  preferred  option  is  poor,  or  by  reconsidering 
the  assessment  of  the  performance  of  one  or  more  options  on  an  attribute. 

Advisory  prompts  will  notify  users  when  an  option  is  itself  dominated,  or 
is  especially  poor  in  some  dimension.  In  addition,  the  system  will  track 
user  efforts  to  establish  a  dominance  structure,  and  prompt  when  altera¬ 
tions  in  previous  preferences  are  excessive  according  to  a  user-set 
criterion. 

4. 3. 3. 3  Focus  (planned')  - -Traditional  multiattribute  utility  analysis 
requires  precise  numerical  assessments  of  the  relative  importance  of  all 
evaluative  dimensions.  Both  ELIMINATE  and  JUSTIFY  simplify  the  choice 
process  by  enabling  the  user  to  employ  simple  non-compensatory  judgments, 
i.e.,  comparisons  within  (rather  than  across)  attributes.  An  ordering  of 
attributes  by  importance  is  the  only  across -attribute  information  required. 
The  result  is  a  selection  of  options  that  may,  on  occasion,  overlook  impor¬ 
tant  tradeoffs.  Advisory  prompts  are  provided  to  protect  against  this.  An 
alternative  strategy  for  simplifying  choice,  however,  is  to  retain  the  goal 
of  selecting  alternatives  with  the  highest  expected  utility  (i.e.,  the 
highest  overall  score  allowing  for  tradeoffs),  but  to  elicit  compensatory 
assessments  from  users  only  to  the  degree  required  by  the  particular  choice 
problem  at  hand. 

The  FOCUS  guide  steps  the  user  through  the  minimal  set  of  assessments 
required  to  narrow  the  option  set  to  the  number  desired.  FOCUS  begins  with 
the  weakest  possible  test,  dominance,  to  determine  if  that  is  sufficient  to 
obtain  the  needed  weeding  out.  Only  if  this  falls  does  it  step  progres¬ 
sively  through  queries  for  stronger  and  stronger  inputs  from  the  user.  In 
all  cases,  it  seeks  out  aspects  of  preference  knowledge  that  are  likely  to 
have  the  greatest  impact  on  narrowing  down  the  option  set  .a.g.,  the  order 
of  importance  of  attribute  A  and  B;  whether  C  is  at  least  twice  as  impor¬ 
tant  as  D;  etc.).  As  a  result,  the  user's  choice  problem  may  be  solved 
with  the  least  judgment  effort  from  the  user. 

In  addition,  at  any  point,  the  user  has  the  option  of  short-circuiting  this 
process  by  instructing  the  system  to  "extrapolate"  a  set  of  constraints 
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that  would  solve  the  problem.  The  system  then  selects  a  set  of  weights 
consistent  with  the  Information  thus  far  provided  by  the  user,  and  ranks 
the  alternatives  with  respect  to  those  weights.  A  sensitivity  analysis  Is 
also  displayed,  to  permit  the  user  to  assess  how  sensitive  the  ranking  Is 
to  the  set  of  weights  chosen.  If  he  desires,  the  user  may  then  provide  ad¬ 
ditional  judgments  of  his  own  to  "correct"  the  system's  extrapolation. 


Implementation 


An  implementation  of  the  personnel  selection  aid  is  underway.  This  section 
provides  a  summary  of  the  hardware  and  software  approach  being  taken. 


4.4.1  Hardware .  At  present  an  IBM  AT  system  is  b^ing  used  for  development 
of  the  aid.  Software  produced  will  operate  on  both  PC-  and  AT-compatible 
machines.  These  are  both  16-bit  microprocessors,  the  PC  having  an  8 -bit 
data  path  provided  by  the  Intel  8088  processor,  and  the  AT  having  a  16 -bit 
data  path  due  to  its  Intel  80286  processor.  For  development  purposes, 
machines  utilized  will  be  equipped  with  floating  point  co-processors 
(8087/80287)  to  improve  computational  speed,  hard  disk.,  and  at  least  512KB 
of  random  access  memory.  Software  to  be  developed  will  not  require  the  co¬ 
processors  (but  will  utilize  them  when  available)  and  will  require  no  more 
than  256KB  of  memory  (the  minimum  available  on  the  PC  or  AT). 


The  aiding  system  planned,  like  the  submarine  approach/attack  planning  aid 
developed  under  Phase  I  of  this  project,  will  utilize  interactive  graphics 
heavily.  We  anticipate  that  the  present  low- resolution  (320  x  200  pixels 
with  4  simultaneous  colors)  video  systems  used  by  personal  computer  owners 
will  be  largely  replaced  within  the  next  two  to  three  years  with  higher 
resolution  systems.  For  this  reason,  we  are  utilizing  the  recently  avail¬ 
able  IBM  Enhanced  Graphics  Adapter  (640  x  350  pixels  with  16  simultaneous 
colors)  and  an  RGB  color  monitor  of  no  less  than  640  x  350  pixel  resolution 


We  plan  to  continue  using  a  mouse-based  input  device  to  minimize  reliance 
on  keyboard  input,  but  will  develop  a  parallel  approach  that  will  permit 
software  operation  with  either  mouse  or  keyboard.  A  light  pen  is  supported 
as  well. 
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4.4.2  Software  Software  will  consist  of  the  following: 

•  A  set  of  software  modules  written  in  C  and  comprising  the  user- 
system  Interface.  These  will  provide  the  user  link  to  the 
various  personalizing  modules  (Select,  Planning,  Adjust,  Alert, 
Advisory).  These  will  be  as  generic  as  possible- -suitable  for 
rapid  construction  of  personalized  aids  in  other  contexts. 

•  A  Data  or  Knowledge  module  will  consist  of  two  components:  (1) 
a  processing  subsystem  written  in  C  containing  the  rules  ap¬ 
propriate  to  support  evaluations  based  on  a  variety  of 
techniques,  ranging  from  elimination  by  aspects  to  multiat¬ 
tribute  utility  analysis;  (2)  a  database  management  subsystem, 
written  in  C  and  utilizing  in  addition  components  of  an  exist¬ 
ing  system,  dBASE  III. 

•  A  graphics  interface  module.  Because  of  the  current  lack  of 
standards  for  graphics  software  and  the  variety  of  graphics 
hardware  which  must  be  supported  if  reasonable  market  penetra¬ 
tion  is  to  be  achieved,  all  graphics  functions  will  be  per¬ 
formed  by  a  distinct  software  module.  This  will  make  future 
modifications  relatively  simple.  This  is  being  written  in  C. 

At  present,  we  are  utilizing  a  graphics  system  known  as  "Halo" 
and  developed  by  Media  Cybernetics,  Inc.  Programming  of  this 
system  is  somewhat  similar  to  programming  of  the  Virtual  Device 
Interface  (VDI)  graphics  system  being  released  very  shortly  by 
IBM.  As  soon  as  feasible,  we  will  replace  the  Halo  system  with 
the  VDI  system.  The  VDI  approach  offers  the  capability  of 
automatically  utilizing  whatever  resolution  is  available  on  a 
(VDI- supported)  device. 
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5.0  CONCLUSION 


The  work  reported  here  has  demonstrated  that  principles  of  personalized  and 
prescriptive  decision  aiding,  originally  developed  in  the  context  of  sub¬ 
marine  command  and  control,  can  be  generalized  successfully  to  a  quite  dif¬ 
ferent  decision  making  context.  Submarine  attack  planning  and  personnel 
evaluation  differ  in  degree  of  time  stress,  in  the  relative  importance  of 
inference  and  choice,  in  the  availability  of  objective  measures  of  success, 
in  the  organizational  role  of  the  aid  and  the  user,  and  in  numerous  other 
ways.  Yet  each  of  these  contexts  poses  a  similar  requirement  for  decision 
support  that  is  tailored  to  individual  styles  of  problem  solving  and  deci¬ 
sion  making  and  which  provides,  at  the  same  time,  prescriptive  guidance  and 
advice.  A  common  set  of  cognitive  interface  modules  has  been  found  to 
satisfy  this  need  in  each  case. 

Work  to  be  done  in  the  remainder  of  this  project  (Phase  II)  includes  the 
following; 

•  completed  implementation  of  the  prototype  personalized  evalua¬ 
tion  system,  as  planned; 

•  testing  of  the  aid  with  potential  users  in  the  domain  of  per¬ 
sonnel  selection  (test  materials  involving  a  database  of  per¬ 
sonnel  resumes  have  already  been  prepared) ; 

•  demonstration  of  the  system  to  ONR  and  other  interested  parties 

•  exploration  of  Phase  III  support  for  technical  refinement  and 
testing  of  the  aid,  marketing  and  distrioucion,  through  con¬ 
tacts  with  commercial  software  firms  and  investment  sources. 
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